1. Field of the Invention
The present invention concerns a process and a unit for gradual processing and, when necessary, graphic display of progressively coded image data, which help to shorten and, with respect to minimization of system usage, optimize the time span during which network users are provided progressively refined image resolution when gradually downloading image data from a central network server to their local client computers in the form of transferred and decoded partial data quantities as preview images, for example.
2. Description of the Prior Art
The following provides a brief presentation of the principle of progressive image coding as applied within the framework of the Progressive JPEG Standard. Because the solution that forms the basis of the invention is, however, independent of the actual image coding standard and only one progressive process is assumed, other standards, e.g. JPEG 2000 or Interlaced GIF, can be used besides the Progressive JPEG Standard described here.
The file format that has come to be known as “Progressive JPEG” is an expansion of the graphic file format JPEG that can be used to gradually build and save a photographically realistic image in a Web browser. In this process, already downloaded partial data quantities ALi [kByte] are decoded in several decoding steps simultaneously during downloading of the data quantity to be transferred
                              L          total                =                              1            1.024                    ⁢                      MByte            kByte                    ⁢                                    ∑                              i                =                1                            N                        ⁢                          Δ              ⁢                                                          ⁢                                                L                  i                                ⁡                                  [                  MByte                  ]                                                                                        (        1        )            of an image file from a central network server to the local client computer of a network user so that ever more detailed preview images Vi of the graphic to be reconstructed can be displayed until the total image file has been downloaded from the network server. The resolution of the graphic will become gradually ever finer during the loading process and the resulting picture will be sharper until the predefined resolution RB is achieved through the quantization of the original image. An advantage of this process is that a person viewing a Web page with embedded graphic elements will get an initial visual impression of the image file when the download begins and does not have to wait until the complete image has been transferred. The goal is                to shorten the elapsing wait time for the graphic display of useable j-th version of a preview image Vi for creating an early visual impression of the downloaded image file        
                                          T            W                    ⁡                      (            j            )                          ⁢                              :                    =                                    ∑                              i                =                1                            j                        ⁢                          Δ              ⁢                                                          ⁢                                                t                                      w                    ,                    i                                                  ⁡                                  [                  s                  ]                                                                                        (                  2          ⁢          a                )                            so that the network user already at an early point in time ti of the loading process can have valuable image information from the data quantity transferred up to that point        
                                                                        L                ist                            ⁡                              (                j                )                                      ⁢                          :                                =                                                    ∑                                  i                  =                  1                                j                            ⁢                              Δ                ⁢                                                                  ⁢                                  L                  i                                                      =                                                            L                  gas                                ·                1.024                            ⁢                              kByte                MByte                            ⁢                                                ∑                                      i                    =                    1                                    j                                ⁢                                  Δ                  ⁢                                                                          ⁢                                                            l                      i                                        ⁡                                          [                      kByte                      ]                                                                                                          ⁢                                  ⁢                              with            ⁢                                                  ⁢            j                    =                      {                          1              ,                              2                ⁢                                                                  ⁢                …                            ⁢                                                          ,              N                        }                                              (                  2          ⁢          b                )                            made available whereby        
                    Δ        ⁢                                  ⁢                  l          i                ⁢                  :                ⁢                  =                                    100              ·                              1                1.024                                      ⁢                                          MByte                kByte                            ·                                                Δ                  ⁢                                                                          ⁢                                      l                    i                                                                    L                  gas                                                                                        (                  2          ⁢          c                )                            which designates the percentage increase of the partial data quantity received and decoded in the decoding step Di.        to transparently display the progress of the image transfer achieved at each decoding step Di at the time points ti, which shows an improvement of the image quality, and        controls the loading process interactively, if necessary, to cancel or continue.        
In order to achieve this, the images used with progressive JPEG are split up as in the baseline JPEG process in 8×8 blocks and are transformed using the Discrete Cosinus Transformation (DCT). According to the quantification, for which parameters may be set, each block will not be coded immediately but will first be temporarily saved in a data buffer until all blocks contained in the image are quantified. The buffer content will then be coded in several steps. Therefore there is no sequential coding of the individual blocks as in the baseline JPEG process. The loading and decoding of partial data quantities for display of rough preview images with gradually refined resolution proceeds significantly faster then a line-by-line image composition for reconstructing the fine resolution of the original image.
The technical problem encountered in decoding in this context lies in the optimal determination of the time distances of two immediately consecutive decoding steps Di−1 and Di that must be selected in such a way that the mean data reception rate R, the improvements in image quality Q achieved in the individual coding steps and the available performance capacity and the relative load (burden)
                    ρ        =                              100            ·                                          T                D                                            T                W                                              =                      100            ·                                                                                ∑                                          i                      =                      1                                        N                                    ⁢                                      Δ                    ⁢                                                                                  ⁢                                          t                                              D                        ,                        i                                                                                                                                  ∑                                          i                      =                      1                                        N                                    ⁢                                      Δ                    ⁢                                                                                  ⁢                                          t                                              W                        ,                        i                                                                                                        ⁢                                                          [              %              ]                                                          (                  3          ⁢          a                )            of the applied decoding and display system can be taken into account. In this instance,
                                          T            W                    ⁢                      :                          =                                            ∑                              i                =                1                            N                        ⁢                          Δ              ⁢                                                          ⁢                              t                                  W                  ,                  1                                                              =                                    t              N                        -                                          t                0                            ⁢                                                          [              s              ]                                                          (                  3          ⁢          b                )            designates the total required wait time from the beginning of the download process to the time to=0 s until the display of the final version Vn of maximum resolution RB of an image at time tN and
                              T          D                ⁢                              :                    =                                    ∑                              i                =                1                            N                        ⁢                          Δ              ⁢                                                          ⁢                                                t                                      D                    ,                    i                                                  ⁢                                                                  [                s                ]                                                                        (                  3          ⁢          c                )            is the total required time span for decoding and graphic visualization of this final version VN, whereby
                              t          D                ⁢                  ≤          1                ⁢                  T          W                                    (                  3          ⁢          d                )            must be valid; Δtw,i is the wait time between both decoding steps Di−1 and Di, Δthd D,i is the actual required computation time for decoding and graphic visualization of the partial data quantity ΔLi and
                              ρ          i                =                  100          ·                                                    Δ                ⁢                                                                  ⁢                                  t                                      D                    ,                    i                                                                              Δ                ⁢                                                                  ⁢                                  t                                      W                    ,                    i                                                                        ⁢                                                  [            %            }                                              (                  3          ⁢          e                )            the usage of the decoding and display system at the time interval Δtw,i which may not be greater than 100%. The following must also apply
                              Δ          ⁢                                          ⁢                      t                          D              ,              i                                      ⁢                  ≤          1                ⁢                  Δ          ⁢                                          ⁢                      t                          W              ,              i                                ⁢                                          ⁢                      ∀                          i              .                                                          (                  3          ⁢          f                )            Conventional processes used in the current state of technology usual determine the time durations ΔtD,l between the time points ti−1 and ti of sequential decoding steps Di−1 and Di either independently from the received data quantity List (j) (Version 1), the decoding steps are implemented at regular time distances ΔtD (Version 2) or use a combination of both processes (Version 3). As is explained in the following, these methods run into technologically conditioned limits.
When executing version 1, a procedure executed to determine the time distancesΔtD,i≡ΔtW,i:=ti−ti−1≠const.[s] (for iε[1,2, . . . , N}  (4)of sequential decoding processes Di−1 and Di starts the i-th decoding (Di) and display step (Vi), when an established, but variable, data quantity ΔLi of progressively coded image data is available to the decoding and display system. So, as an example, the first decoding step Di is executed after the first block B1 of progressively coded image data is received by the decoding system. The partial data quantity ΔL1 of the first block B1 is thereby an optional system parameter. Further decoding steps D2 through Dn are implemented after further blocks B2 thru Bn, whose sizes ΔL2 through ΔLn depend respectively on the sizes ΔL1 through ΔLN−1 for the previously received blocks B1 through Bn−1 and their system parameters are determined suitable, were received by the decoding system. Such a determination of the decoding distances Δtp,l takes into account the mean transfer rate
                    R        =                              8            ⁢                                          Bit                Byte                            ·                                                L                  ges                                                  T                  W                                                              ⁢                                          ⁢                                          =                      8            ⁢                                          Bit                Byte                            ·                              1                1.024                                      ⁢                                          Mbyte                kByte                            ·                              1                                  T                  W                                            ·                                                ∑                                      i                    =                    1                                    N                                ⁢                                  Δ                  ⁢                                                                          ⁢                                                            L                      i                                        ⁢                                                                                  [                                          Mbit                      ⁢                                              /                                            ⁢                      s                                        ]                                                                                                          (        5        )            the total data quantity received Ltotals during the time Tw only indirectly via the system parameters ΔLi. Then there is a slow reception of the data at long wait times Δtw,l between the decoding steps Di−1 and Di, whereas there is a fast reception of data at very short wait times Δtw,i. In the latter case, the decoding processes can no longer be executed in a timely manner due to the limited performance capacity of the decoding system is some circumstances. In order to avoid this, in many systems the instant value
                                          R            i                    =                      8            ⁢                                          Bit                Byte                            ·                              1                1.024                                      ⁢                                          MByte                kByte                            ·                                                Δ                  ⁢                                                                          ⁢                                      L                    i                                                                    Δ                  ⁢                                                                          ⁢                                      t                                          D                      ,                      i                                                                                                          ⁢                                  ⁢                  (                                    for              ⁢                                                          ⁢              i                        ∈                          (                              1                ,                                  2                  ⁢                                                                          ⁢                  …                                ,                N                            )                                                          (        6        )            of the data rate R of received image data form a system parameter that is either regularly measured or is recognized as an estimate by the decoding system.
If, as in version 2, the display steps Vi are executed in regular, constant time intervals Δtp, the total image data quantity received at the time tj of display by the decoding and display system
                                                                                          L                  ist                  ′                                ⁡                                  (                  j                  )                                            :=                                                                    j                    ·                    Δ                                    ⁢                                                                          ⁢                  L                                =                                  1.024                  ⁢                                                                          ⁢                                                                                    k                        ⁢                                                                                                  ⁢                        Byte                                            MByte                                        ·                    j                    ·                                          L                      ges                                        ·                    Δ                                    ⁢                                                                          ⁢                                      l                    ⁢                                                                                  [                                          k                      ⁢                                                                                          ⁢                      Byte                                        ]                                                                                                                                          (                                                      with                    ⁢                                                                                  ⁢                    j                                    ∈                                      {                                          1                      ,                      2                      ,                      …                      ⁢                                                                                          ,                      N                                        }                                                  )                            ,                                                          (                  7          ⁢          a                )            whereby
                              Δ          ⁢                                          ⁢          l                :=                              100            ·                          1              1.024                                ⁢                                          ⁢                                    MByte                              k                ⁢                                                                  ⁢                Byte                                      ·                                                            Δ                  ⁢                                                                          ⁢                  L                                                  L                  ges                                            ⁢                                                          [              %              ]                                                          (                  7          ⁢          b                )            with ΔL1=ΔL2= . . . ΔLi= . . . =ΔLN=:ΔL [kByte]  (7c)
the increase as a percentage of the received data and designated in the individual decoding steps as decoded constant partial data quantities ΔL, will be decoded and graphically visualized. The time intervalstD,i:=ti−ti−1=const. [s] (for iε{1,2, . . . , N})  (8a)between two consecutive decoding steps Di−1 and Di, wherebyΔtD,1=ΔtD,2= . . . ΔtD,i= . . . ΔtD,N=:ΔtD[s] and ΔtD,1≦ΔtW,1  (8b)is valid, form a system parameter of the decoding system and also take into account the performance capacity of the decoding system. In the case of such a manner of proceeding, the data rate
                                                                        R                I                            =                              8                ⁢                                                      Bit                    Byte                                    ·                                      1                    1.024                                                  ⁢                                                                  ⁢                                                                            M                      ⁢                                                                                          ⁢                      Byte                                                              k                      ⁢                                                                                          ⁢                      Byte                                                        ·                                                            Δ                      ⁢                                                                                          ⁢                                              L                        i                                                                                    Δ                      ⁢                                                                                          ⁢                                              t                                                  W                          ,                          i                                                                                                                                                                                                                                        ⁢                              =                                                      8                    ⁢                                                                  Bit                        Byte                                            ·                                              1                        1.024                                                              ⁢                                                                                  ⁢                                                                                            M                          ⁢                                                                                                          ⁢                          Byte                                                                          k                          ⁢                                                                                                          ⁢                          Byte                                                                    ·                                                                        Δ                          ⁢                                                                                                          ⁢                                                      L                            i                                                                                                    Δ                          ⁢                                                                                                          ⁢                                                      t                            W                                                                                              ·                                              N                        N                                                                              =                                                            8                      ⁢                                                                        Bit                          Byte                                                ·                                                                              L                            ges                                                                                T                            W                                                                                                                =                                          R                      ⁢                                                                                          [                                              M                        ⁢                                                                                                  ⁢                        Bit                        ⁢                                                  /                                                ⁢                        s                                            ]                                                                                                                                              (                                                with                  ⁢                                                                          ⁢                  i                                ∈                                  {                                      1                    ,                    2                    ,                    …                    ⁢                                                                                  ,                    N                                    }                                            )                                                          (        9        )            of the received data will implicitly be taken into account. At a small data rate Ri of a received partial data quantity ΔLi, the visual improvement of the displayed image between decoding steps Di−1 and Di is small, whereas a high data rate Ri of the received partial data quantity ΔLi leads to a dramatic improvement ΔQi of the image quality Q. An overload due to the limited performance capacity of the decoding system is hereby excluded. There is a disadvantage, however, in that the usage p of the decoding system remains constant with respect to time independently of the data rates Ri of the received partial data quantities ΔLi
A combination of both manners of proceeding according to version 3 leads to usage of the decoding system that can be modified with respect to time.
The usage pi is dependent on the partial data quantity ΔLi received in the time interval Δtw,i.
Common to all three versions, however, is the fact that statistical and visual properties of a transferred image cannot be taken into account. It may therefore occur that consecutive display steps Vi−1 and Vi can lead to no discernable improvement of the image resolution RB,l to the viewer.
FIG. 1 shows a typical course of image quality Q depending on the percentage
                    l        :                  100          ·                      1            1.024                    ·                                    M              ⁢                                                          ⁢              Byte                                      k              ⁢                                                          ⁢              Byte                                ·                                    L                              L                ges                                      ⁢                                                  [            %            ]                                              (        10        )            of the received data quantity L [kByte] of progressively coded images compressed in the JPEG 2000 format referring to the total data quantity Ltotal to be transferred. As a statistical quality metric
                              Q          1                :                  100          ·                                    (                              1                -                                                      e                    i                                                        e                    0                                                              )                        ⁢                                                  [            %            ]                                              (                  11          ⁢          a                )            for the current image resolution RB,i is, in this instance, the mean square error (MSE)
                                                                        e                1                            :=                                                1                                                            μ                      max                                        ·                                          v                      max                                                                      ·                                                      ∑                                          μ                      =                      1                                                              μ                      max                                                        ⁢                                                                          ⁢                                                            ∑                                              v                        =                        1                                                                    v                        max                                                              ⁢                                                                                  ⁢                                                                  (                                                                              v                                                          n                              ,                                                              μ                                ⁢                                                                                                                                  ⁢                                v                                                                                                              -                                                      v                                                          i                              ,                                                              μ                                ⁢                                                                                                                                  ⁢                                v                                                                                                                                    )                                            2                                                                                                                                              (                                                for                  ⁢                                                                          ⁢                  i                                ∈                                  {                                      1                    ,                    2                    ,                    …                    ⁢                                                                                  ,                    N                                    }                                            )                                                          (                  11          ⁢          b                )            between the final version Vn of an image to be reconstructed whose image quality Qn ideally corresponds to the image qualityQorig:=100%  (11c)of the transferred original image Vorig, and forms the basis of the respectively viewed version of a preview image Vi at lower resolution and has been normalized to a quality range between 0% and 100%. In this case, VN,uv designates the pixel value of the original image Vorig to be transferred for the pixel (μ,v), Vi,uv the pixel value of the i-th preview image Vi for the pixel (μ, v) and
                                          e            0                    ≡                      e            max                          :=                              1                                          μ                max                            ·                              v                max                                              ·                                    ∑                              μ                =                1                                            μ                max                                      ⁢                                                  ⁢                                          ∑                                  v                  =                  1                                                  v                  max                                            ⁢                                                          ⁢                              V                                  N                  ,                                      μ                    ⁢                                                                                  ⁢                    v                                                  2                                                                        (                  11          ⁢          d                )            the maximum possible error. A quality metric of 100% provides the best available quality that can be achieved when the total image data quantity Ltotal has been completely and successfully transferred. In this case (i=N) the mean square error ei is ideally equal to zero:eN=0.  (11e)An image quality of 0% exists when not image data has been transferred. In this case, (i=0) the mean square error ei achieves a maximum value of emax.
FIG. 2 shows another diagram for displaying a typical course of the statistical image quality Q depending on the percentage l of the received data quantity L of progressively coded image data compressed in the JPEG 2000 format referring to the total data quantity Ltotal (in MByte) to be transferred. Here, in addition to the continual function course Q(l), the time ti of the decoding procedures Di with the associated percentage of the data quantity share Δll and image quality values AQi for a statistically equal quality improvementΔQi:=h(Δli)=Qi−Qi−1(for iε{1,2, . . . N,})  (12)of 10.00% per decoding step i has been specified, whereby both relationshipsQi:+Q(li) and  (12a)Qi=1:=Q(li−1)  (12b)give the image quality of the preview images Vi and Vi−1. This shows that when there is a small portion Δll the quality increase AQi is high, i.e. the data quantity ΔLi that must be received between two improvement steps Di−1 and Di is initially relatively small and increases with the increase of the existing data quantity List(j) at the point in time tj.
As can be seen in FIG. 2, a large number N of decoding steps Di for a proportionally small quantity Δll of data within the framework of the determination of the decoding step Di described above in accordance with version 1 leads to an overload of the decoding system.
In practice, the courses of the image quality Q illustrated in FIGS. 1 and 2 result from statistically mean and normalized values of a random quality metric. The relationship indicated above between the image quality Q and the mean square error e is therefore understood for purely illustrative purposes. In fact, the normalized image quality courses Q(l) are stored in a data memory and are determined independently of the transfer.